An Endorsement-based Approach to Student Modeling for Planner-controlled Tutors

This paper describes an approach to student modeling for intelligent tutoring systems based on an explicit representation of the tutor's beliefs about the student and the arguments for and against those beliefs (called endorsements). A lexicographic comparison of arguments, sorted according to evidence reliability, provides a principled means of determining those beliefs that are considered true, false, or uncertain. Each of these beliefs is ultimately justif ied by underlying assessment data. The endorsement-based approach to student modeling is particularly appropriate for tutors controlled by instructional planners. These tutors place greater demands on a student model than opportunistic tutors. Numeric calculi approaches are less well-suited because it is diff icult to correctly assign numbers for evidence reliability and rule plausibility. It may also be difficult to interpret final results and provide suitable combining functions. When numeric measures of uncertainty are used, arbitrary numeric thresholds are often required for planning decisions. Such an approach is inappropriate when robust context-sensitive planning decisions must be made. Instead, the ability to examine beliefs and justifications is required. This paper presents a TMS-based implementation of the endorsement-based approach to student modeling, discusses the advantages of this approach for plannercontrolled tutors, and compares this approach to alternatives. 1. Introduct ion — limitations of numeric student models This paper proposes a symbolic (i.e., non-numeric) means of coping with uncertainty in student modeling. Rather than represent the uncertainty of the tutor's beliefs with numeric degrees of confidence the student model explicitly records arguments (called endorsements in [Cohen, 85]) for and against each belief. No interpretation of numbers or use This research was sponsored by the Armstrong Laboratory, Human Resource Directorate (formerly AFHRL / ID I ) , in cooperation with NASA under subcontract #063, RICIS research activity #ET.24 (NASA Cooperative Agreement NCC9-16). of numeric combining functions is required. Instead, the different kinds of arguments are compared based on the reliability of their evidence to decide if belief or disbelief in a proposition is justified. Previous research on the Blackboard Instructional Planner [Murray, 90], a planner-controlled tutor for leaching troubleshooting for a complex hydraulic-electronicmechanical device, illustrated some of the shortcomings of numeric student models. That research motivates the research presented here. Before reviewing the earlier research, we briefly consider the role and demands placed on the student model in both planning and non-planning (i.e., opportunistic) tutors. In opportunistic tutors the student model may be used to decide what skills to address (e.g., WEST [Burton and Brown, 82]) or what topics to explore (e.g., MENOTUTOR [Woolf, 84]). Other uses are problem selection (e.g., BIP [Barr, 76]) or hint generation (e.g., WUSOR-I1 [Carr, 77]). Frequently, diagnostic student modeling is used to provide a detailed model of a student's problem solving and to evaluate its correctness (e.g., PROUST [Johnson, 86]). The student model for a planner-controlled tutor must not only address these issues but others. A sophisticated student model is needed to allow customized plan generation based on an initial assessment of the student. Later it is needed to track and help revise the plan as instruction is delivered. It must interpret different kinds of assessments (student data) such as the student's background, any student selfassessment, test questions, any instructor assessment, student-initiated questions, and student problem-solving actions. Typically, the student model for opportunistic intelligent tutoring systems wi l l handle a much more l imited range of assessment data and have fewer responsibilities. For example, those tutors that act as problem-solving monitors (the most common paradigm) predominantly focus on assessing problem-solving actions for hint generation and future problem selection (e.g., IMTS [Towne et a/., 89]). The student model of the Blackboard Instructional Planner illustrates some of the shortcomings of numeric student models and how they can l imit tutor capabilities. That student model is an overlay [Carr and Goldstein, 77] of a semantic net representation of domain concepts. Associated 1100 Principles of A! Applications with each concept is a number representing the tutor's confidence that the student has acquired the concept. The numbers are initialized from a pre-instruction questionnaire according to inferred cognitive stereotypes [Rich, 79] and later are adjusted according to the student's test and problemsolving performance. With this numeric approach the tutor tended to either replan at the wrong times or not replan when it should. The problem was that planning decisions could only rely on these numbers, which were compared to threshold values. Replanning can easily go awry because of the difficulty of determining precisely how to adjust the numeric weights to integrate the different kinds of assessment data, and because of the arbitrary nature of the three planning thresholds that were used. One threshold measured when a concept was learned, another when it was forgotten, and a third when an instructional activity was making insufficient progress. When the thresholds and updates were adjusted conservatively the planner tended not to replan when it should. When they were adjusted less conservatively the planner tended to replan when it should not. These problems led to the development of an endorsement-based student model (ESM). The remainder of this paper describes the endorsement-based approach, compares it to alternatives, and argues that it is particularly appropriate for planner-controlled tutors. 2. The endorsement-based approach to student modeling The key aspects of the ESM arc as follows: /. Explicit representation of tutor beliefs and their endorsements—propositions represent the tutor's beliefs about the student's skills along with arguments for and against those beliefs. 2. Inheritance of endorsements—an ISA hierarchy represents the subject matter. The ESM uses the hierarchy to represent the degree to which a student has generalized a skill. Endorsements for a generic skill (a skill that can be applied to all members of a class) are inherited down the hierarchy towards subclasses (or instances) representing more specific ski l ls. Endorsements against a generic skill are propagated up towards superclasses representing more general skills. 3. Wide variety of assessments—several different kinds of information, varying both in specificity, source, and reliability are incorporated. 4. Lexicographic comparison of arguments— endorsements are sorted into equivalence classes according to re l iabi l i ty . This ordering allows lexicographic comparison of pro and con arguments. The result of the comparison is a label for each belief — be l i eved t rue , be l ievedfa lse, u n k n o w n (no data), or uncertain — and an indication of the decisive argument, if any, that indicates how well justified a belief is. 5. Consistency between endorsements and labels—the student model explicitly represents the justification for each endorsement and tutor belief. A l l justifications are ultimately grounded in assessments (student data). If endorsements become invalid or labels change then consistency is maintained between derived endorsements and any labels that depend on them. These features are best illustrated by examples. 2.1 Examples of endorsement-based s tudent mode l ing This section presents a scenario demonstrating the endorsement-based approach. Assume the student is learning to troubleshoot a device and must first learn how the device and its individual parts operate. Figure 1 shows a class hierarchy of parts of the device. Classes of parts are connected to subclasses by solid arrows. These in turn are connected to part instances by dashed arrows. The tutor's goal is to ensure that the student understands the operation of all of the device's hydraulic valves. This goal (a generic skill) is represented by the proposition SK(op, hydraulic valves). SK stands for "studeni knows" (a notation adopted from IPeachcy and McCalla, 86]). The general form is SK(skill, node) where node is either a class or instance. SK (op, LJVK4) is believed true when the tutor believes the student understands the operation of the UVK4 valve. SK(op, latchable valves) is believed true when the tutor believes the student understands the operation of all the latchable valves: UVK4, UVK9, and UVK10. So, if SK(op, UVK4) was believed false then SK(op, latchable valves) would also have to be believed false. The scenario below illustrates how an endorsement-based student modeling system can cope with several different kinds of assessments, can infer new beliefs based on inheritance using the links in Figure 1, and can retract beliefs that are no longer justified. It also shows how pro and con arguments are compared.